Publications (detailed list) - NNML Laboratory

This page contains the titles and abstracts of papers written by members of the BYU Neural Networks and Machine Learning (NNML) Research Group. Postscript files are available for most papers. A more concise list is available.

DMP3: A Dynamic Multi-Layer Perceptron Construction Algorithm

Authors: Tim L. Andersen and Tony R. Martinez
Reference: International Journal of Neural Systems, volume 2, pages 145–166, 2001.
BibTeX:
@article{AndersenIJNS,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {{DMP3}: A Dynamic Multi-Layer Perceptron Construction Algorithm},
journal = {International Journal of Neural Systems},
volume = {2},
pages = {145--166},
year = {2001},
}
Download the file: pdf

Optimal Artificial Neural Network Architecture Selection for Voting

Authors: Tim L. Andersen and Michael E. Rimer and Tony R. Martinez
Abstract: This paper studies the performance of standard architecture selection strategies, such as cost/performance and CV based strategies, for voting methods such as bagging. It is shown that standard architecture selection strategies are not optimal for voting methods and tend to underestimate the complexity of the optimal network architecture, since they only examine the performance of the network on an individual basis and do not consider the correlation between responses from multiple networks.
Reference: In Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN’01, pages 790–795, 2001.
BibTeX:
@inproceedings{andersen.ijcnn01.oas,
author = {Andersen, Tim L. and Rimer, Michael E. and Martinez, Tony R.},
title = {Optimal Artificial Neural Network Architecture Selection for Voting},
booktitle = {Proceedings of the {IEEE} International Joint Conference on Neural Networks {IJCNN}'01},
pages = {790--795},
year = {2001},
}
Download the file: pdf

Optimal Artificial Neural Network Architecture Selection for Bagging

Authors: Tim L. Andersen and Tony R. Martinez
Abstract: This paper studies the performance of standard architecture selection strategies, such as cost/performance and CV based strategies, for voting methods such as bagging. It is shown that standard architecture selection strategies are not optimal for voting methods and tend to underestimate the complexity of the optimal network architecture, since they only examine the performance of the network on an individual basis and do not consider the correlation between responses from multiple networks.
Reference: In Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN’01, pages 790–795, 2001.
BibTeX:
@inproceedings{andersen_2001b,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Optimal Artificial Neural Network Architecture Selection for Bagging},
booktitle = {Proceedings of the {IEEE} International Joint Conference on Neural Networks {IJCNN}'01},
pages = {790--795},
year = {2001},
}
Download the file: pdf

The Little Neuron that Could

Authors: Tim L. Andersen and Tony R. Martinez
Reference: In Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN’99, CD paper #191, 1999.
BibTeX:
@inproceedings{andersen.ijcnn1999.wag,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {The Little Neuron that Could},
booktitle = {Proceedings of the {IEEE} International Joint Conference on Neural Networks {IJCNN}'99, {CD} paper #191},
year = {1999},
}
Download the file: pdf

Cross Validation and MLP Architecture Selection

Authors: Tim L. Andersen and Tony R. Martinez
Reference: In Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN’99, CD paper #192, 1999.
BibTeX:
@inproceedings{andersen.ijcnn99.cv,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Cross Validation and {MLP} Architecture Selection},
booktitle = {Proceedings of the {IEEE} International Joint Conference on Neural Networks {IJCNN}'99, {CD} paper #192},
year = {1999},
}
Download the file: pdf, ps

Constructing Higher Order Perceptrons with Genetic Algorithms

Authors: Tim L. Andersen and Tony R. Martinez
Reference: In Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN’98, pages 1920–1925, 1998.
BibTeX:
@inproceedings{andersen.ijcnn1998,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Constructing Higher Order Perceptrons with Genetic Algorithms},
booktitle = {Proceedings of the {IEEE} International Joint Conference on Neural Networks {IJCNN}'98},
pages = {1920--1925},
year = {1998},
}
Download the file: pdf

Wagging: A learning approach which allows single layer perceptrons to outperform more complex learning algorithms

Authors: Tim L. Andersen and Tony R. Martinez
Reference: In Submitted to IEEE Transactions on Neural Networks, 1997.
BibTeX:
@inproceedings{andersen.ieee99.wag,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Wagging: A learning approach which allows single layer perceptrons to outperform more complex learning algorithms},
booktitle = {Submitted to {IEEE} Transactions on Neural Networks},
year = {1997},
}
Download the file: ps

Genetic Algorithms and Higher Order Perceptron Networks

Authors: Tim L. Andersen and Tony R. Martinez
Reference: In Proceedings of the International Workshop on Neural Networks and Neurocontrol, pages 217–223, 1997.
BibTeX:
@inproceedings{andersen.sian97,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Genetic Algorithms and Higher Order Perceptron Networks},
booktitle = {Proceedings of the International Workshop on Neural Networks and Neurocontrol},
pages = {217--223},
year = {1997},
}
Download the file: pdf

Using Multiple Node Types to Improve the Performance of DMP (Dynamic Multilayer Perceptron)

Authors: Tim L. Andersen and Tony R. Martinez
Abstract: This paper discusses a method for training multi-layer perceptron networks called DMP2 (Dynamic Multi-layer Perceptron 2). The method is based upon a divide and conquer approach which builds networks in the form of binary trees, dynamically allocating nodes and layers as needed. The focus of this paper is on the effects of using multiple node types within the DMP framework. Simulation results show that DMP2 performs favorably in comparison with other learning algorithms, and that using multiple node types can be beneficial to network performance.
Reference: In Proceedings of the IASTED International Conference on Artificial Intelligence, Expert Systems and Neural Networks, pages 249–252, 1996.
BibTeX:
@inproceedings{andersen.iasted96.dmp2,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Using Multiple Node Types to Improve the Performance of {DMP} (Dynamic Multilayer Perceptron)},
booktitle = {Proceedings of the {IASTED} International Conference on Artificial Intelligence, Expert Systems and Neural Networks},
pages = {249--252},
year = {1996},
}
Download the file: ps

The Effect of Decision Surface Fitness on Dynamic Multi-layer Perceptron Networks

Authors: Tim L. Andersen and Tony R. Martinez
Abstract: The DMP1 (Dynamic Multi-layer Perceptron 1) network training method is based upon a divide and conquer approach which builds networks in the form of binary trees, dynamically allocating nodes and layers as needed. This paper introduces the DMP1 method, and compares the preformance of DMP1 when using the standard delta rule training method for training individual nodes against the performance of DMP1 when using a genetic algorithm for training. While the basic model does not require the use of a genetic algorithm for training individual nodes, the results show that the convergence properties of DMP1 are enhanced by the use of a genetic algorithm with an appropriate fitness function.
Reference: In Proceedings of the World Congress on Neural Networks , pages 177–181, 1996.
BibTeX:
@inproceedings{andersen.wcnn96.dmp_ga,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {The Effect of Decision Surface Fitness on Dynamic Multi-layer Perceptron Networks},
booktitle = {Proceedings of the World Congress on Neural Networks },
pages = {177--181},
year = {1996},
}
Download the file: ps

Learning and Generalization with Bounded Order Rule Sets

Authors: Tim L. Andersen
Abstract: This thesis deals with the problem of inducing useful rules, or extracting critical, higher-order conjunctions of attributes, from a set of preclassified examples, where little or nothing is known about the underlying functional form of the distribution from which the examples were taken. The approach taken in this thesis differs from that normally used in that it does not limit the size of the rule search space. Rather, every possible conjunction of input attributes is considered by the learning algorithm as a potential rule component. In so doing, this thesis is attempting to determine (mainly from an empirical standpoint) how generalization performance is affected when certain areas of the search space are ignored, as compared to when the entire search space is considered. In dealing with the above question, this thesis studies several methods for inducing rules and using them for classification of novel examples. This thesis also uses results obtained with the C4.5 rule-induction method for comparison purposes, and to support the main points of the thesis. The results show that higher-order rules are not required to approximate many real world learning problems. In addition, the difficulty of generating optimal rule sets is discussed, where the measure of optimality is the complexity or size of the rule set and/or the degree of predictive accuracy, and the problem of NP-completeness is discussed in relation to these two optimality measures.
Reference: Master’s thesis, Brigham Young University, April 1995.
BibTeX:
@mastersthesis{andersen_95a,
author = {Andersen, Tim L.},
title = {Learning and Generalization with Bounded Order Rule Sets},
school = {Brigham Young University},
month = {April},
year = {1995},
}
Download the file: ps

Learning and Generalization with Bounded Order Rule Sets

Authors: Tim L. Andersen and Tony R. Martinez
Abstract: All current rule-based methods found in the literature use some form of heuristic(s) in order to limit the size of the rule search space examined by the learning algorithm. This paper is an attempt to determine (mainly from an empirical standpoint) how generalization performance is affected when certain areas of the rule search space are ignored, as compared to when the entire search space is considered. This is done by exhaustively generating all rules for several small real-world problems and then determining how accuracy decreases as the size of the search space is iteratively reduced. The results show that higher-order rules are not required to approximate many real world learning problems. In dealing with the above question, several methods for inducing rules and using them for classification of novel examples are tested.
Reference: In Proceedings of the 10th International Symposium on Computer and Information Sciences, pages 419–426, 1995.
BibTeX:
@inproceedings{andersen_95b,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Learning and Generalization with Bounded Order Rule Sets},
booktitle = {Proceedings of the 10th International Symposium on Computer and Information Sciences},
pages = {419--426},
year = {1995},
}
Download the file: ps

NP-Completeness of Minimum Rule Sets

Authors: Tim L. Andersen and Tony R. Martinez
Abstract: Rule induction systems seek to generate rule sets which are optimal in the complexity of the rule set. This paper develops a formal proof of the NP-Completeness of the problem of generating the simplest rule set (MIN RS) which accurately predicts examples in the training set for a particular type of generalization algorithm and complexity measure. The proof is then informally extended to cover a broader spectrum of complexity measures and learning algorithms.
Reference: In Proceedings of the 10th International Symposium on Computer and Information Sciences, pages 411–418, 1995.
BibTeX:
@inproceedings{andersen_95c,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {{NP}-Completeness of Minimum Rule Sets},
booktitle = {Proceedings of the 10th International Symposium on Computer and Information Sciences},
pages = {411--418},
year = {1995},
}
Download the file: ps

A Provably Convergent Dynamic Training Method for Multilayer Perceptron Networks

Authors: Tim L. Andersen and Tony R. Martinez
Abstract: This paper presents a new method for training multi-layer perceptron networks called DMP1 (Dynamic Multi-layer Perceptron 1). The method is based upon a divide and conquer approach which builds networks in the form of binary trees, dynamically allocating nodes and layers as needed. The individual nodes of the network are trained using a gentetic algorithm. The method is capable of handling real-valued inputs and a proof is given concerning its convergence properties of the basic model. Simulation results show that DMP1 performs favorably in comparison with other learning algorithms.
Reference: In Proceedings of the 2nd International Symposium on Neuroinformatics and Neurocomputers, pages 77–84, 1995.
BibTeX:
@inproceedings{andersen_95d,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {A Provably Convergent Dynamic Training Method for Multilayer Perceptron Networks},
booktitle = {Proceedings of the 2nd International Symposium on Neuroinformatics and Neurocomputers},
pages = {77--84},
year = {1995},
}
Download the file: ps

Learning and Generalization with Bounded Order Critical Feature Sets

Authors: Tim L. Andersen and Tony R. Martinez
Abstract: It is the case that many real world learning problems exhibit a great deal of regularity. It is likely that the solutions to learning problems which exhibit such regularity can be approximated utilizing only simple (low-order) features gathered from analysis of pre-classified examples. However, little specific work has been done to demonstrate the utility of low-order features. This paper presents methods for gathering low order features from an existing data set of preclassified examples, and using these features for classification of novel patterns from the problem domain. It then conducts experiments using the methods presented on several real world classification problems and reports the results. The results show that pattern classification methods involving low-order feature sets have promise and warrant further research.
Reference: In Proceedings of the AI’93 Australian Joint Conference on Artificial Intelligence, page 450, 1993.
BibTeX:
@inproceedings{andersen_93a,
author = {Andersen, Tim L. and Martinez, Tony R.},
title = {Learning and Generalization with Bounded Order Critical Feature Sets},
booktitle = {Proceedings of the {AI}'93 Australian Joint Conference on Artificial Intelligence},
pages = {450},
year = {1993},
}
Download the file: ps

Efficient Construction of Networks for Learned Representations with General to Specific Relationships

Authors: J. Cory Barker and Tony R. Martinez
Abstract: Machine learning systems often represent concepts or rules as sets of attribute-value pairs. Many learning algorithms generalize or specialize these concept representations by removing or adding pairs. Thus concepts are created that have general to specific relationships. This paper presents algorithms to connect concepts into a network based on their general to specific relationships. Since any concept can access related concepts quickly, the resulting structure allows increased efficiency in learning and reasoning. The time complexity of one set of learning models improves from O(n log n) to O(log n) (where n is the number of nodes) when using the general to specific structure.
Reference: Yfantis, Evangelos A., editor, Intelligent Systems, volume 1, pages 617–625, Kluwer Academic Publishers, 1995.
BibTeX:
@article{barker_95a,
author = {Barker, J. Cory and Martinez, Tony R.},
title = {Efficient Construction of Networks for Learned Representations with General to Specific Relationships},
editor = {Yfantis, Evangelos A.},
journal = {Intelligent Systems},
volume = {1},
pages = {617--625},
publisher = {Kluwer Academic Publishers},
year = {1995},
}
Download the file: ps

Eclectic Machine Learning

Authors: J. Cory Barker
Abstract: This dissertation presents a family of inductive learning systems that derive general rules from specific examples. These systems combine the benefits of neural networks, ASOCS, and symbolic learning algorithms. The systems presented here learn incrementally with good speed and generalization. They are based on a parallel architectural model that adapts to the problem being learned. Learning is done without requiring user adjustment of sensitive parameters, and noise is tolerated with graceful degradation in performance. The systems described in this work are based on features. Features are subsets of the input space. One group of learning algorithms begins with general features and specializes those features to match the problem that is being learned. Another group creates specific features and then generalizes those features. The final group combines the approaches used in the first two groups to gain the benefits of both. The algorithms are O(m log m), where m is the number of nodes in the network, and the number of inputs and output values are treated as constants. An enhanced network topology reduces time complexity to O(log m). Empirical results show that the algorithms give good generalization and that learning converges in a small number of training passes.
Reference: PhD thesis, Brigham Young University, February 1994.
BibTeX:
@phdthesis{barker_diss,
author = {Barker, J. Cory},
title = {Eclectic Machine Learning},
school = {Brigham Young University},
month = {February},
year = {1994},
}
Download the file: ps

Proof of Correctness for ASOCS AA3 Networks

Authors: J. Cory Barker and Tony R. Martinez
Abstract: This paper analyzes Adaptive Algorithm 3 (AA3) of Adaptive Self-Organizing Concurrent Systems (ASOCS) and proves that AA3 correctly fulfills the rules presented. Several different models for ASOCS have been developed. AA3 uses a distributed mechanism for implementing rules so correctness is not obvious. An ASOCS is an adaptive network composed of many simple computing elements operating in parallel. An ASOCS operates in one of two modes: learning and processing. In learning mode, rules are presented to the ASOCS and incorporated in a self-organizing fashion. In processing mode, the ASOCS acts as a parallel hardware circuit that performs the function defined by the learned rules.
Reference: IEEE Transactions on Systems, Man, and Cybernetics, volume 3, pages 503–510, 1994.
BibTeX:
@article{barker_94a,
author = {Barker, J. Cory and Martinez, Tony R.},
title = {Proof of Correctness for {ASOCS} {AA3} Networks},
journal = {{IEEE} Transactions on Systems, Man, and Cybernetics},
volume = {3},
pages = {503--510},
year = {1994},
}
Download the file: ps

Generalization by Controlled Expansion of Examples

Authors: J. Cory Barker and Tony R. Martinez
Abstract: SG (Specific to General) is a learning system that derives general rules from specific examples. SG learns incrementally with good speed and generalization. The SG network is built of many simple nodes that adapt to the problem being learned. Learning is done without requiring user adjustment of sensitive parameters and noise is tolerated with graceful degradation in performance. Nodes learn important features in the input space and then monitor the ability of the features to predict output values. Learning is O(n log n) for each example, where n is the number of nodes in the network, and the number of inputs and output values are treated as constants. An enhanced network topology reduces time complexity to O(log n). Empirical results show that the model gives good generalization and that learning converges in a small number of training passes.
Reference: In Proceedings of The Seventh International Symposium on Artificial Intelligence, pages 142–149, 1994.
BibTeX:
@inproceedings{barker_94b,
author = {Barker, J. Cory and Martinez, Tony R.},
title = {Generalization by Controlled Expansion of Examples},
booktitle = {Proceedings of The Seventh International Symposium on Artificial Intelligence},
pages = {142--149},
year = {1994},
}
Download the file: ps

GS: A Network that Learns Important Features

Authors: J. Cory Barker and Tony R. Martinez
Abstract: GS is a network for supervised inductive learning from examples that uses ideas from neural networks and symbolic inductive learning to gain benefits of both methods. The network is built of many simple nodes that learn important features in the input space and then monitor the ability of the features to predict output values. The network avoids the exponential nature of the number of features by using information gained by general features to guide the creation of more specific features. Empirical evaluation of the model on real world data has shown that the network provides good generalization performance. Convergence is accomplished within a small number of training passes. The network provides these benefits while automatically allocating and deleting nodes and without requiring user adjustment of any parameters. The network learns incrementally and operates in a parallel fashion.
Reference: In Proceedings of The World Congress on Neural Networks, volume 3, pages 376–380, July 1993.
BibTeX:
@inproceedings{barker_93c,
author = {Barker, J. Cory and Martinez, Tony R.},
title = {{GS}: A Network that Learns Important Features},
booktitle = {Proceedings of The World Congress on Neural Networks},
volume = {3},
pages = {376--380},
month = {July},
year = {1993},
}
Download the file: ps

Generalization by Controlled Intersection of Examples

Authors: J. Cory Barker and Tony R. Martinez
Abstract: SG (Specific to General) is a network for supervised inductive learning from examples that uses ideas from neural networks and symbolic inductive learning to gain benefits of both methods. The network is built of many simple nodes that learn important features in the input space and then monitor the ability of the features to predict output values. The network avoids the exponential nature of the number of features by creating specific features for each example and then expanding those features; making them more general. Expansion of a feature terminates when it encounters another feature with contradicting outputs. Empirical evaluation of the model on real-world data has shown that the network provides good generalization performance. Convergence is accomplished within a small number of training passes. The network provides these benefits while automatically allocating and deleting nodes and without requiring user adjustment of any parameters. The network learns incrementally and operates in a parallel fashion.
Reference: In Proceedings of The Sixth Australian Joint Conference on Artificial Intelligence, pages 323–327, 1993.
BibTeX:
@inproceedings{barker_93a,
author = {Barker, J. Cory and Martinez, Tony R.},
title = {Generalization by Controlled Intersection of Examples},
booktitle = {Proceedings of The Sixth Australian Joint Conference on Artificial Intelligence},
pages = {323--327},
year = {1993},
}
Download the file: ps

Learning and Generalization Controlled by Contradiction

Authors: J. Cory Barker and Tony R. Martinez
Abstract: One page overview of the SG (Specific to General) learning model.
Reference: In Proceedings of The International Conference on Artificial Neural Networks, 1993.
BibTeX:
@inproceedings{barker_93b,
author = {Barker, J. Cory and Martinez, Tony R.},
title = {Learning and Generalization Controlled by Contradiction},
booktitle = {Proceedings of The International Conference on Artificial Neural Networks},
year = {1993},
}
Download the file: ps

Extending ID3 through Discretization of Continuous Inputs

Authors: Rick Bertelsen and Tony R. Martinez
Abstract: This paper presents a mechanism to extend ID3 by classifying real valued inputs. Real valued inputs are classified through a neural network model termed the Competitive Classifier (CC). The CC forwards discrete classification results to the ID3 system, and accepts feedback from the ID3 system. Through the use of feedback, the ID3 system guides the CC into improving classifications.
Reference: In Proceedings of FLAIRS’94 Florida Artificial Intelligence Research Symposium, pages 122–125, 1994.
BibTeX:
@inproceedings{bertelsen_94,
author = {Bertelsen, Rick and Martinez, Tony R.},
title = {Extending {ID3} through Discretization of Continuous Inputs},
booktitle = {Proceedings of {FLAIRS}'94 Florida Artificial Intelligence Research Symposium},
pages = {122--125},
year = {1994},
}
Download the file: ps

Automatic Feature Extraction in Machine Learning

Authors: Rick Bertelsen
Abstract: This thesis presents a machine learning model capable of extracting discrete classes out of continuous valued input features. This is done using a neurally inspired novel competitive classifier (CC) which feeds the discrete classifications forward to a supervised machine learning model. The supervised learning model uses the discrete classifications and perhaps other information available to solve a problem. The supervised learner then generates feedback to guide the CC into potentially more useful classifications of the continuous valued input features. Two supervised learning models are combined with the CC creating ASOCS-AFE and ID3-AFE. Both models are simulated and the results are analyzed. Based on these results, several areas of future research are proposed.
Reference: Master’s thesis, Brigham Young University, 1994.
BibTeX:
@mastersthesis{bertelsen_th,
author = {Bertelsen, Rick},
title = {Automatic Feature Extraction in Machine Learning},
school = {Brigham Young University},
year = {1994},
}
Download the file: ps

Improved Hopfield Nets by Training with Noisy Data

Authors: Fred Clift and Tony R. Martinez
Reference: In Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN’01, pages 1138–1143, 2001.
BibTeX:
@inproceedings{Cliftijcnn2001,
author = {Clift, Fred and Martinez, Tony R.},
title = {Improved Hopfield Nets by Training with Noisy Data},
booktitle = {Proceedings of the {IEEE} International Joint Conference on Neural Networks {IJCNN}'01},
pages = {1138--1143},
year = {2001},
}
Download the file: pdf

Search Techniques for Fourier-Based Learning

Authors: Adam Drake and Dan Ventura
Abstract: Fourier-based learning algorithms rely on being able to efficiently find the large coefficients of a function’s spectral representation. In this paper, we introduce and analyze techniques for finding large coefficients. We show how a previously introduced search technique can be generalized from the Boolean case to the real-valued case, and we apply it in branch-and-bound and beam search algorithms that have significant advantages over the best-first algorithm in which the technique was originally introduced.
Reference: In Proceedings of the AAAI Workshop on Search in Artificial Intelligence and Robotics, 2008.
BibTeX:
@inproceedings{drake2008a,
author = {Drake, Adam and Ventura, Dan},
title = {Search Techniques for {F}ourier-Based Learning},
booktitle = {Proceedings of the {AAAI} Workshop on Search in Artificial Intelligence and Robotics},
year = {2008},
}
Download the file: pdf

Sentiment Regression: Using Real-Valued Scores to Summarize Overall Document Sentiment

Authors: Adam Drake and Eric Ringger and Dan Ventura
Abstract: In this paper, we consider a sentiment regression problem: summarizing the overall sentiment of a review with a real-valued score. Empirical results on a set of labeled reviews show that real-valued sentiment modeling is feasible, as several algorithms improve upon baseline performance. We also analyze performance as the granularity of the classification problem moves from two-class (positive vs. negative) towards infinite-class (real-valued).
Reference: In Proceedings of the IEEE International Conference on Semantic Computing, 2008.
BibTeX:
@inproceedings{drv.icsc2008,
author = {Drake, Adam and Ringger, Eric and Ventura, Dan},
title = {Sentiment Regression: Using Real-Valued Scores to Summarize Overall Document Sentiment},
booktitle = {Proceedings of the {IEEE} International Conference on Semantic Computing},
year = {2008},
}
Download the file: pdf

Comparing High-Order Boolean Features

Authors: Adam Drake and Dan Ventura
Abstract: Many learning algorithms attempt, either explicitly or implicitly, to discover useful high-order features. When considering all possible functions that could be encountered, no particular type of high-order feature should be more useful than any other. However, this paper presents arguments and empirical results that suggest that for the learning problems typically encountered in practice, some high-order features may be more useful than others.
Reference: In Proceedings of the Joint Conference on Information Sciences, pages 428–431, July 2005.
BibTeX:
@inproceedings{drake.ventura.jcis2005,
author = {Drake, Adam and Ventura, Dan},
title = {Comparing High-Order Boolean Features},
booktitle = {Proceedings of the Joint Conference on Information Sciences},
pages = {428--431},
month = {July},
year = {2005},
}
Download the file: pdf

A Practical Generalization of Fourier-Based Learning

Authors: Adam Drake and Dan Ventura
Abstract: This paper presents a search algorithm for finding functions that are highly correlated with an arbitrary set of data. The functions found by the search can be used to approximate the unknown function that generated the data. A special case of this approach is a method for learning Fourier representations. Empirical results demonstrate that on typical real-world problems the most highly correlated functions can be found very quickly, while combinations of these functions provide good approximations of the unknown function.
Reference: In ICML ’05: Proceedings of the 22nd International Conference on Machine Learning, pages 185–192, New York, NY, USA, 2005. ACM Press.
BibTeX:
@inproceedings{drake.ventura.icml2005,
author = {Drake, Adam and Ventura, Dan},
title = {A Practical Generalization of Fourier-Based Learning},
booktitle = {ICML '05: Proceedings of the 22nd International Conference on Machine Learning},
pages = {185--192},
publisher = {ACM Press},
address = {New York, NY, USA},
year = {2005},
}
Download the file: pdf

Predicting and Preventing Coordination Problems in Cooperative Learning Systems

Authors: Nancy Fulda and Dan Ventura
Abstract: We present a conceptual framework for creating Q-learning-based algorithms that converge to optimal equilibria in cooperative multiagent settings. This framework includes a set of conditions that are sufficient to guarantee optimal system performance. We demonstrate the efficacy of the framework by using it to analyze several well-known multi-agent learning algorithms and conclude by employing it as a design tool to construct a simple, novel multiagent learning algorithm.
Reference: In Proceedings of the International Joint Conference on Artificial Intelligence, page to appear, Hyderabad, India, January 2007.
BibTeX:
@inproceedings{fulda.ijcai07,
author = {Fulda, Nancy and Ventura, Dan},
title = {Predicting and Preventing Coordination Problems in Cooperative Learning Systems},
booktitle = {Proceedings of the International Joint Conference on Artificial Intelligence},
pages = {to appear},
address = {Hyderabad, India},
month = {January},
year = {2007},
}
Download the file: pdf

Learning a Rendezvous Task with Dynamic Joint Action Perception

Authors: Nancy Fulda and Dan Ventura
Abstract: Groups of reinforcement learning agents interacting in a common environment often fail to learn optimal behaviors. Poor performance is particularly common in environments where agents must coordinate with each other to receive rewards and where failed coordination attempts are penalized. This paper studies the effectiveness of the Dynamic Joint Action Perception (DJAP) algorithm on a grid-world rendezvous task with this characteristic. The effects of learning rate, exploration strategy, and training time on algorithm effectiveness are discussed. An analysis of the types of tasks for which DJAP learning is appropriate is also presented.
Reference: In Proceedings of the International Joint Conference on Neural Networks, pages 627–632, Vancouver, BC, July 2006.
BibTeX:
@inproceedings{fulda.ijcnn06,
author = {Fulda, Nancy and Ventura, Dan},
title = {Learning a Rendezvous Task with Dynamic Joint Action Perception},
booktitle = {Proceedings of the International Joint Conference on Neural Networks},
pages = {627--632},
address = {Vancouver, BC},
month = {July},
year = {2006},
}
Download the file: pdf

Incremental Policy Learning: An Equilibrium Selection Algorithm for Reinforcement Learning Agents with Common Interests

Authors: Nancy Fulda and Dan Ventura
Abstract: We present an equilibrium selection algorithm for reinforcement learning agents that incrementally adjusts the probabilityof executing each action based on the desirability of the outcome obtained in the last time step. The algorithm assumes that at least one coordination equilibrium exists and requires that the agents have a heuristic for determining whether or not the equilibrium was obtained. In deterministic environments with one or more strict coordination equilibria, the algorithm will learn to play an optimal equilibrium as long as the heuristic is accurate. Empirical data demonstrate that the algorithm is also effective in stochastic environments and is able to learn good joint policies when the heuristic’s parameters are estimated during learning, rather than known in advance.
Reference: In Proceedings of the International Joint Conference on Neural Networks, pages 1121–1126, July 2004.
BibTeX:
@inproceedings{fulda.ijcnn04,
author = {Fulda, Nancy and Ventura, Dan},
title = {Incremental Policy Learning: An Equilibrium Selection Algorithm for Reinforcement Learning Agents with Common Interests},
booktitle = {Proceedings of the International Joint Conference on Neural Networks},
pages = {1121--1126},
month = {July},
year = {2004},
}
Download the file: pdf

Target Sets: A Tool for Understanding and Predicting the Behavior of Interacting Q-learners

Authors: Nancy Fulda and Dan Ventura
Abstract: Reinforcement learning agents that interact in a common environment frequently affect each others’ perceived transition and reward distributions. This can result in convergence of the agents to a sub-optimal equilibrium or even to a solution that is not an equilibrium at all. Several modifications to the Q-learning algorithm have been proposed which enable agents to converge to optimal equilibria under specified conditions. This paper presents the concept of target sets as an aid to understanding why these modifications have been successful and as a tool to assist in the development of new modifications which are applicable in a wider range of situations.
Reference: In Proceedings of the Joint Conference on Information Sciences, pages 1549–1552, September 2003.
BibTeX:
@inproceedings{fulda.jcis03,
author = {Fulda, Nancy and Ventura, Dan},
title = {Target Sets: A Tool for Understanding and Predicting the Behavior of Interacting Q-learners},
booktitle = {Proceedings of the Joint Conference on Information Sciences},
pages = {1549--1552},
month = {September},
year = {2003},
}
Download the file: pdf

Concurrently Learning Neural Nets: Encouraging Optimal Behavior in Reinforcement Learning Systems.

Authors: Nancy Fulda and Dan Ventura
Reference: In IEEE International Workshop on Soft Computing Techniques in Instrumentation, Measurement, and Related Applications (SCIMA), May 2003.
BibTeX:
@incollection{fulda_2003a,
author = {Fulda, Nancy and Ventura, Dan},
title = {Concurrently Learning Neural Nets: Encouraging Optimal Behavior in Reinforcement Learning Systems.},
booktitle = {{IEEE} International Workshop on Soft Computing Techniques in Instrumentation, Measurement, and Related Applications ({SCIMA})},
month = {May},
year = {2003},
}
Download the file: ps, pdf

Dynamic Joint Action Perception for Q-Learning Agents.

Authors: Nancy Fulda and Dan Ventura
Reference: In To Appear in Proceedings of the 2003 International Conference on Machine Learning and Applications, Los Angeles, CA, 2003.
BibTeX:
@inproceedings{fulda_2003b,
author = {Fulda, Nancy and Ventura, Dan},
title = {Dynamic Joint Action Perception for Q-Learning Agents.},
booktitle = {To Appear in Proceedings of the 2003 International Conference on Machine Learning and Applications, Los Angeles, {CA}},
year = {2003},
}
Download the file: ps, pdf

Towards Automatic Shaping in Robot Navigation.

Authors: Todd S. Peterson and Nancy Owens and James L. Carroll
Reference: In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), 2001.
BibTeX:
@inproceedings{fulda_2001a,
author = {Peterson, Todd S. and Owens, Nancy and Carroll, James L.},
title = {Towards Automatic Shaping in Robot Navigation.},
booktitle = {Proceedings of the {IEEE} International Conference on Robotics and Automation ({ICRA})},
year = {2001},
}
Download the file: ps, pdf

Memory-guided Exploration in Reinforcement Learning.

Authors: James L. Carroll and Todd S. Peterson and Nancy Owens
Reference: In Proceedings of the INNS-IEEE International Joint Conference on Neural Networks (IJCNN), 2001.
BibTeX:
@inproceedings{fulda_2001b,
author = {Carroll, James L. and Peterson, Todd S. and Owens, Nancy},
title = {Memory-guided Exploration in Reinforcement Learning.},
booktitle = {Proceedings of the {INNS}-{IEEE} International Joint Conference on Neural Networks ({IJCNN})},
year = {2001},
}
Download the file: ps, pdf

Using a Reinforcement Learning Controller to Overcome Simulator/Environment Discrepancies.

Authors: Nancy Owens and Todd S. Peterson
Reference: In Proceedings of the IEEE Conference on Systems, Man, and Cybernetics, 2001.
BibTeX:
@inproceedings{fulda_2001c,
author = {Owens, Nancy and Peterson, Todd S.},
title = {Using a Reinforcement Learning Controller to Overcome Simulator/Environment Discrepancies.},
booktitle = {Proceedings of the {IEEE} Conference on Systems, Man, and Cybernetics},
year = {2001},
}
Download the file: pdf

Iterative Non-linear Dimensionality Reduction with Manifold Sculpting

Authors: Michael S. Gashler and Dan Ventura and Tony Martinez
Abstract: Many algorithms have been recently developed for reducing dimensionality by projecting data onto an intrinsic non-linear manifold. Unfortunately, existing algorithms often lose significant precision in this transformation. Manifold Sculpting is a new algorithm that iteratively reduces dimensionality by simulating surface tension in local neighborhoods. We present several experiments that show Manifold Sculpting yields more accurate results than existing algorithms with both generated and natural data-sets. Manifold Sculpting is also able to benefit from both prior dimensionality reduction efforts.
Reference: In Platt, J.C. and Koller, D. and Singer, Y. and Roweis, S., editor, Advances in Neural Information Processing Systems 20, pages 513–520, MIT Press, Cambridge, MA, 2008.
BibTeX:
@incollection{gashler2007nips,
author = {Gashler, Michael S. and Ventura, Dan and Martinez, Tony},
title = {Iterative Non-linear Dimensionality Reduction with Manifold Sculpting},
editor = {Platt, J.C. and Koller, D. and Singer, Y. and Roweis, S.},
booktitle = {Advances in Neural Information Processing Systems 20},
pages = {513--520},
publisher = {MIT Press},
address = {Cambridge, MA},
year = {2008},
}
Download the file: pdf

Learning by Discrimination: A Constructive Incremental Approach

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: This paper presents i-AA1*, a constructive, incremental learning algorithm for a special class of weightless, self-organizing networks. In i-AA1*, learning consists of adapting the nodes’ functions and the network’s overall topology as each new training pattern is presented. Provided the training data is consistent, computational complexity is low and prior factual knowledge may be used to “prime” the network and improve its predictive accuracy and/or efficiency. Empirical generalization results on both toy problems and more realistic tasks demonstrate promise.
Reference: Journal of Computers, volume 2 (7), pages 49–58, September 2007.
BibTeX:
@article{cgc.jcp2007,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {Learning by Discrimination: A Constructive Incremental Approach},
journal = {Journal of Computers},
volume = {2},
number = {7},
pages = {49--58},
month = {September},
year = {2007},
issn = {1796-203X},
}
Download the file: pdf

A Constructive Incremental Learning Algorithm for Binary Classification Tasks

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Reference: In Proceedings of SMCals/06, pages 213–218, 2006.
BibTeX:
@inproceedings{cgc_smc2006,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {A Constructive Incremental Learning Algorithm for Binary Classification Tasks},
booktitle = {Proceedings of SMCals/06},
pages = {213--218},
year = {2006},
}
Download the file: pdf

An Efficient Metric for Heterogeneous Inductive Learning Applications in the Attribute-Value Language

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: Many inductive learning problems can be expressed in the classical attribute-value language. In order to learn and to generalize, learning systems often rely on some measure of similarity between their current knowledge base and new information. The attribute-value language defines a heterogeneous multi-dimensional input space, where some attributes are nominal and others linear. Defining similarity, or proximity, of two points in such input spaces is non trivial. We discuss two representative homogeneous metrics and show examples of why they are limited to their own domains. We then address the issues raised by the design of a heterogeneous metric for inductive learning systems. In particular, we discuss the need for normalization and the impact of don’t-care values. We propose a heterogeneous metric and evaluate it empirically on a simplified version of ILA.
Reference: Yfantis, Evangelos A., editor, Intelligent Systems), volume 1, pages 341–350, Kluwer Academic Publishers, 1995.
BibTeX:
@article{cgc_94c,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {An Efficient Metric for Heterogeneous Inductive Learning Applications in the Attribute-Value Language},
editor = {Yfantis, Evangelos A.},
journal = {Intelligent Systems)},
volume = {1},
pages = {341--350},
publisher = {Kluwer Academic Publishers},
year = {1995},
}
Download the file: ps

An Integrated Framework for Learning and Reasoning

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: Learning and reasoning are both aspects of what is considered to be intelligence. Their studies within AI have been separated historically, learning being the topic of machine learning and neural networks, and reasoning falling under classical (or sym b olic) AI. However, learning and reasoning are in many ways interdependent. This paper discusses the nature of some of these interdependencies, and proposes a general framework called FLARE, that combines inductive learning using prior knowledge together with reasoning. Several examples are presented that serve as a benchmark to test the framework, including classical induction, several commonsense protocols, and the use of reasoning to discover prior knowledge that can be used as a learning bias for inductive learning.
Reference: Journal of Artificial Intelligence Research, volume 3, pages 147–185, 1995.
BibTeX:
@article{cgc_95a,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {An Integrated Framework for Learning and Reasoning},
journal = {Journal of Artificial Intelligence Research},
volume = {3},
pages = {147--185},
year = {1995},
}
Download the file: ps

AA1*: A Dynamic Incremental Network that Learns by Discrimination

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: An incremental learning algorithm for a special class of self-organising, dynamic networks is presented. Learning is effected by adapting both the function performed by the nodes and the overall network topology, so that the network grows (or shrinks) over time to fit the problem. Convergence is guaranteed on any arbitrary Boolean dataset and empirical generalisation results demonstrate promise.
Reference: In Proceedings of the International Conference on Artificial Neural Networks and Genetic Algorithms (ICANNGA’95), pages 45–48, 1995.
BibTeX:
@inproceedings{cgc_95b,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {{AA1}*: A Dynamic Incremental Network that Learns by Discrimination},
booktitle = {Proceedings of the International Conference on Artificial Neural Networks and Genetic Algorithms ({ICANNGA}'95)},
pages = {45--48},
year = {1995},
}
Download the file: ps

Analysis of the Convergence and Generalization of AA1

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: AA1 is an incremental learning algorithm for Adaptive Self-Organizing Concurrent Systems (ASOCS). ASOCS are self-organizing, dynamically growing networks of computing nodes. AA1 learns by discrimination and implements knowledge in a distributed fashion over all the nodes. This paper reviews AA1 from the perspective of convergence and generalization. A formal proof that AA1 converges on any arbitrary Boolean instance set is given. A discussion of generalization and other aspects of AA1, including the problem of handling inconsistency, follows. Results of simulations with real-world data are presented. They show that AA1 gives promising generalization.
Reference: Journal of Parallel and Distributed Computing, volume 26, pages 125–131, 1995.
BibTeX:
@article{cgc_95c,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {Analysis of the Convergence and Generalization of {AA1}},
journal = {Journal of Parallel and Distributed Computing},
volume = {26},
pages = {125--131},
year = {1995},
}
Download the file: ps

On Integrating Inductive Learning with Prior Knowledge and Reasoning

Authors: Christophe Giraud-Carrier
Abstract: Learning and reasoning are both aspects of what is considered to be intelligence. Their studies within AI have been separated historically, learning being the topic of neural networks and machine learning, and reasoning falling under classical (or symbolic) AI. However, learning and reasoning share many interdependencies, and the integration of the two may lead to more powerful models. This dissertation examines some of these interdependencies, and describes several models, culminating in a system called FLARE (Framework for Learning And REasoning). The proposed models integrate inductive learning with prior knowledge and reasoning. Learning is incremantal, prior knowledge is given by a teacher or deductively obtained by instantiating commonsense knowledge, and reasoning is non-monotonic. Simulation results on several datasets and classical commonsense protocols demonstrate promise.
Reference: PhD thesis, Brigham Young University, December 1994.
BibTeX:
@phdthesis{cgc_diss,
author = {Giraud-Carrier, Christophe},
title = {On Integrating Inductive Learning with Prior Knowledge and Reasoning},
school = {Brigham Young University},
month = {December},
year = {1994},
}
Download the file: ps

Seven Desirable Properties for Artificial Learning Systems

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: Much effort has been devoted to understanding learning and reasoning in artificial intelligence, giving rise to a wide collection of models. For the most part, these models focus on some observed characteristic of human learning, such as induction or analogy, in an effort to emulate (and possibly exceed) human abilities. We propose seven desirable properties for artificial learning systems: incrementality, non-monotonicity, inconsistency and conflicting defaults handling, abstraction, self- organization, generalization, and computational tractability. We examine each of these properties in turn and show how their (combined) use can improve learning and reasoning, as well as potentially widen the range of applications of artificial learning systems. An overview of the algorithm PDL2, that begins to integrate the above properties, is given as a proof of concept.
Reference: In Proceedings of FLAIRS’94 Florida Artificial Intelligence Research Symposium, pages 16–20, 1994.
BibTeX:
@inproceedings{cgc_94a,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {Seven Desirable Properties for Artificial Learning Systems},
booktitle = {Proceedings of {FLAIRS}'94 Florida Artificial Intelligence Research Symposium},
pages = {16--20},
year = {1994},
}
Download the file: ps

An Incremental Learning Model for Commonsense Reasoning

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: A self-organizing incremental learning model that attempts to combine inductive learning with prior knowledge and default reasoning is described. The inductive learning scheme accounts for useful generalizations and dynamic priority allocation, and effectively supplements prior knowledge. New rules may be created and existing rules modified, thus allowing the system to evolve over time. By combining the extensional and intensional approaches to learning rules, the model remains self-adaptive, while not having to unnecessarily suffer from poor (or atypical) learning environments. By combining rule-based and similarity-based reasoning, the model effectively deals with many aspects of brittleness.
Reference: In Proceedings of the Seventh International Symposium on Artificial Intelligence (ISAI’94), pages 134–141, 1994.
BibTeX:
@inproceedings{cgc_94b,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {An Incremental Learning Model for Commonsense Reasoning},
booktitle = {Proceedings of the Seventh International Symposium on Artificial Intelligence ({ISAI}'94)},
pages = {134--141},
year = {1994},
}
Download the file: ps

Using Precepts to Augment Training Set Learning

Authors: Christophe Giraud-Carrier and Tony R. Martinez
Abstract: The goal of learning systems is to generalize. Generalization is commonly based on the set of criticalfeatures the system has available. Training set learners typically extract critical features from a random set of examples. While this approach is attractive, it suffers from the exponential growth of the number of features to be searched. We propose to extend it by endowing the system with some a priori knowledge, in the form of precepts. Advantages of the augmented system are speed-up, improved generalization, and greater parsimony. This paper presents a precept-driven learning algorithm. Its main features include: 1) distributed implementation, 2) bounded learning and execution times, and 3) ability to handle both correct and incorrect precepts. Results of simulations on real-world data demonstrate promise.
Reference: In Proceedings of the First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems (ANNES’93), pages 46–51, November 1993.
BibTeX:
@inproceedings{cgc_93a,
author = {Giraud-Carrier, Christophe and Martinez, Tony R.},
title = {Using Precepts to Augment Training Set Learning},
booktitle = {Proceedings of the First New Zealand International Two-Stream Conference on Artificial Neural Networks and Expert Systems ({ANNES}'93)},
pages = {46--51},
month = {November},
year = {1993},
}
Download the file: ps

A Precept-Driven Learning Algorithm

Authors: Christophe Giraud-Carrier
Abstract: Machine learning is an attempt at devising mechanisms that machines can use to learn, rather than being explicitly programmed for, real-world applications. The goal of learning systems is to generalize. Generalization is based on the set of critical features the system has available. Training set learners typically extract critical features from a random set of examples drawn from experimentation. This approach can beneficially be extended by endowing the system with some a priori knowledge, in the form of precepts. Advantages of the augmented system include speed-up, improved generalization and greater parsimony. This thesis presents a precept-driven learning algorithm. The main characteristics of the algorithm include: 1) neurally inspired architecture, 2) bounded learning and execution times, and 3) ability to handle both correct and incorrect precepts. Results of simulations on real-world data demonstrate promise.
Reference: Master’s thesis, Brigham Young University, April 1993.
BibTeX:
@mastersthesis{cgc_th,
author = {Giraud-Carrier, Christophe},
title = {A Precept-Driven Learning Algorithm},
school = {Brigham Young University},
month = {April},
year = {1993},
}
Download the file: ps

Spatiotemporal Pattern Recognition in Liquid State Machines

Authors: Eric Goodman and Dan Ventura
Abstract: The applicability of complex networks of spiking neurons as a general purpose machine learning technique remains open. Building on previous work using macroscopic exploration of the parameter space of an (artificial) neural microcircuit, we investigate the possibility of using a liquid state machine to solve two real-world problems: stockpile surveillance signal alignment and spoken phoneme recognition.
Reference: In Proceedings of the International Joint Conference on Neural Networks, pages 7979–7584, Vancouver, BC, July 2006.
BibTeX:
@inproceedings{goodman.ijcnn06,
author = {Goodman, Eric and Ventura, Dan},
title = {Spatiotemporal Pattern Recognition in Liquid State Machines},
booktitle = {Proceedings of the International Joint Conference on Neural Networks},
pages = {7979--7584},
address = {Vancouver, BC},
month = {July},
year = {2006},
}
Download the file: pdf

Effectively Using Recurrently Connected Spiking Neural Networks

Authors: Eric Goodman and Dan Ventura
Abstract: Recurrently connected spiking neural networks are difficult to use and understand because of the complex nonlinear dynamics of the system. Through empirical studies of spiking networks, we deduce several principles which are critical to success. Network parameters such as synaptic time delays and time constants and the connection probabilities can be adjusted to have a significant impact on accuracy. We show how to adjust these parameters to fit the type of problem.
Reference: In Proceedings of the International Joint Conference on Neural Networks, pages 1542–1547, July 2005.
BibTeX:
@inproceedings{goodman.ijcnn05,
author = {Goodman, Eric and Ventura, Dan},
title = {Effectively Using Recurrently Connected Spiking Neural Networks},
booktitle = {Proceedings of the International Joint Conference on Neural Networks},
pages = {1542--1547},
month = {July},
year = {2005},
}
Download the file: pdf

Time Invariance and Liquid State Machines

Authors: Eric Goodman and Dan Ventura
Abstract: Time invariant recognition of spatiotemporal patterns is a common task of signal processing. The liquid state machine (LSM) is a paradigm which robustly handles this type of classification. Using an artificial dataset with target pattern lengths ranging from 0.1 to 1.0 seconds, we train an LSM to find the start of the pattern with a mean absolute error of 0.18 seconds. Also, LSMs can be trained to identify spoken digits, 1-9, with an accuracy of 97.6%, even with scaling by factors ranging from 0.5 to 1.5.
Reference: In Proceedings of the Joint Conference on Information Sciences, pages 420–423, July 2005.
BibTeX:
@inproceedings{goodman.jcis05,
author = {Goodman, Eric and Ventura, Dan},
title = {Time Invariance and Liquid State Machines},
booktitle = {Proceedings of the Joint Conference on Information Sciences},
pages = {420--423},
month = {July},
year = {2005},
}
Download the file: pdf

Extending ASOCS to Training-Set-Style Data

Authors: Edward F. Hart
Abstract: This thesis studies extensions to help the basic ASOCS neural network handle training set data. Background is discussed on the need for neural networks. Instance set math for the ASOCS model is presented. Six data sets are gathered and mapped into the ASOCS neural net. Tests are done with the data sets to assist in the discovery of several problems. Some of these problems have solutions suggested in this thesis.

Mutual exclusion, arbitrary discretization, One Difference minimization, noisy and missing data, generalization using a Hamming distance metric are all discussed during the research section. As a result of these extensions to the basic ASOCS model, training-set-style data is recognized much better than without the extensions. Further research using large test sets, other generalization techniques, and different discretization techniques is suggested.
Reference: Master’s thesis, Brigham Young University, August 1992.
BibTeX:
@mastersthesis{hart_th,
author = {Hart, Edward F.},
title = {Extending {ASOCS} to Training-Set-Style Data},
school = {Brigham Young University},
month = {August},
year = {1992},
}
Download the file: ps

Constructing Low-Order Discriminant Neural Networks Using Statistical Feature Selection

Authors: Eric Henderson and Tony R. Martinez
Reference: Journal of Intelligent Systems, volume 14, 2005.
BibTeX:
@article{henderson.jis2005,
author = {Henderson, Eric and Martinez, Tony R.},
title = {Constructing Low-Order Discriminant Neural Networks Using Statistical Feature Selection},
journal = {Journal of Intelligent Systems},
volume = {14},
year = {2005},
}
Download the file: pdf

Pair Attribute Learning: Network Construction Using Pair Features

Authors: Eric Henderson and Tony R. Martinez
Abstract: We present the Pair Attribute Learning (PAL) algorithm for the selection of relevant inputs and network topology. Correlations on training instance pairs are used to drive network construction of a single-hidden layer MLP. Results on nine learning problems demonstrate 70% less complexity, on average, without a significant loss of accuracy.
Reference: In Proceedings of the IEEE International Joint Conference on Neural Networks IJCNN’02, pages 2556–2561, 2002.
BibTeX:
@inproceedings{EricIJCNN,
author = {Henderson, Eric and Martinez, Tony R.},
title = {Pair Attribute Learning: Network Construction Using Pair Features},
booktitle = {Proceedings of the {IEEE} International Joint Conference on Neural Networks {IJCNN}'02},
pages = {2556--2561},
year = {2002},
}
Download the file: pdf

Prioritized Rule Systems

Authors: Brent W. Hughes
Abstract: Non-von Neumann architectures attempt to overcome the “word-at-a-time” bottleneck of traditional computing systems. Neural nets are a class of non-von architectures whose goal is, among other things, to learn input-output mappings using highly distributed processing and memory. A neural net consists of many simple processing elements (nodes) with modifiable links between them, allowing for a high degree of parallelism. A typical neural net has fixed a topology. It learns by modifying the “weights” or “conductances” of the links between nodes.

Another model with similar goals, called ASOCS, learns by modifying its topology. Unlike a typical neural net, ASOCS is guaranteed to be able to represent and learn any desired mapping and to do so efficiently. This thesis presents an extension of ASOCS called Prioritized Rules Systems. The PRS abstract model provides a foundation for various architectural models that have a number of advantages over other ASOCS models. One example of such an architectural model is presented in the thesis along with a description of a program written to simulate that model. The processing and learning algorithms of the architectural model are based on theorems proved in the thesis.
Reference: Master’s thesis, Brigham Young University, November 1989.
BibTeX:
@mastersthesis{hughes_th,
author = {Hughes, Brent W.},
title = {Prioritized Rule Systems},
school = {Brigham Young University},
month = {November},
year = {1989},
}
Download the file: ps

Improved Backpropagation Learning in Neural Networks with Windowed Momentum

Authors: Butch Istook and Tony R. Martinez
Abstract: Backpropagation, which is frequently used in Neural Network training, often takes a great deal of time to converge on an acceptable solution. Momentum is a standard technique that is used to speed up convergence and maintain generalization performance. In this paper we present the Windowed momentum algorithm, which increases speedup over standard momentum. Windowed momentum is designed to use a fixed width history of recent weight updates for each connection in a neural network. By using this additional information, Windowed momentum gives significant speed-up over a set of applications with same or improved accuracy. Windowed Momentum achieved an average speed-up of 32% in convergence time on 15 data sets, including a large OCR data set with over 500,000 samples. In addition to this speedup, we present the consequences of sample presentation order. We show that Windowed momentum is able to overcome these effects that can occur with poor presentation order and still maintain its speed-up advantages.
Reference: International Journal of Neural Systems, volume 3&4, pages 303–318, 2002.
BibTeX:
@article{IstookIJNS,
author = {Istook, Butch and Martinez, Tony R.},
title = {Improved Backpropagation Learning in Neural Networks with Windowed Momentum},
journal = {International Journal of Neural Systems},
volume = {3&4},
pages = {303--318},
year = {2002},
}
Download the file: pdf

Improving Text Classification using Conceptual and Contextual Features

Authors: Lee S. Jensen and Tony R. Martinez
Abstract: The exponential growth of text available on the Internet has created a critical need for accurate, fast, and general-purpose text classification algorithms. This paper examines the improvement of broadly based text classification by using features that are easily extracted from training documents. These features represent both the conceptual and contextual properties of a target class, and include synonyms, hypernyms, term frequency, and bigrams of nouns, synonyms and hypernyms. Multiple permutations of the features are applied to three different classification models (Coordinate matching, TF*IDF, and naive Bayes) over three diverse data sets (Reuters, USENET, and folk songs). The findings are also compared to previously published results for the rule-based learner Ripper and results obtained by using another naive Bayes classifier, Rainbow. Suggestions are made about how to automatically determine which features to use, based upon the data set in question. The results demonstrate that the introduction of both conceptual and contextual features decreases the error by as much as 33%.
Reference: , pages 101–102, KDD 2000, Text Mining Workshop, Boston. 2000.
BibTeX:
@misc{jensen,
author = {Jensen, Lee S. and Martinez, Tony R.},
title = {Improving Text Classification using Conceptual and Contextual Features},
pages = {101--102},
howpublished = {{KDD} 2000, Text Mining Workshop, Boston.},
year = {2000},
}
Download the file: ps

A Data-dependent Distance Measure for Transductive Instance-based Learning

Authors: Jared Lundell and Dan Ventura
Abstract: We consider learning in a transductive setting using instance-based learning (k-NN) and present a method for constructing a data-dependent distance “metric” using both labeled training data as well as available unlabeled data (that is to be classified by the model). This new data-driven measure of distance is empirically studied in the context of various instance-based models and is shown to reduce error (compared to traditional models) under certain learning conditions. Generalizations and improvements are suggested.
Reference: In Proceedings of the IEEE International Conference on Systems, Man and Cybernetics, pages 2825–2830, October 2007.
BibTeX:
@inproceedings{lundell.smc07,
author = {Lundell, Jared and Ventura, Dan},
title = {A Data-dependent Distance Measure for Transductive Instance-based Learning},
booktitle = {Proceedings of the {IEEE} International Conference on Systems, Man and Cybernetics},
pages = {2825--2830},
month = {October},
year = {2007},
}
Download the file: pdf

Priority ASOCS

Authors: Tony R. Martinez and Brent W. Hughes and Douglas M. Campbell
Abstract: This paper presents an ASOCS (Adaptive Self-Organizing Concurrent System) model for massively parallel processing of incrementally defined rule systems in such areas as adaptive logic, robotics, logical inference, and dynamic control. An ASOCS is an adaptive network composed of many simple computing elements operating asynchronously and in parallel. An ASOCS can operate in either a data processing mode or a learning mode. During data processing mode, an ASOCS acts as a parallel hardware circuit. During learning mode, an ASOCS incorporates a rule expressed as a Boolean conjunction in a distributed fashion in time logarithmic in the number of rules. This paper proposes a learning algorithm and architecture for Priority ASOCS. This new ASOCS model uses rules with priorities. The new model has significant learning time and space complexity improvements over previous models.
Reference: Journal of Artificial Neural Networks , volume 3, pages 403–429, 1994.
BibTeX:
@article{martinez_94a,
author = {Martinez, Tony R. and Hughes, Brent W. and Campbell, Douglas M.},
title = {Priority {ASOCS}},
journal = {Journal of Artificial Neural Networks },
volume = {3},
pages = {403--429},
year = {1994},
}
Download the file: ps

A Generalizing Adaptive Discriminant Network

Authors: Tony R. Martinez and J. Cory Barker and Christophe Giraud-Carrier
Abstract: This paper overviews the AA1 (Adaptive Algorithm 1) model of ASOCS the (Adaptive Self-Organizing Concurrent Systems) approach. It also presents promising empirical generalization results of AA1 with actual data. AA1 is a topologically dynamic network which grows to fit the problem being learned. AA1 generalizes in a self-organizing fashion to a network which seeks to find features which discriminate between concepts. Convergence to a training set is both guaranteed and bounded linearly in time.
Reference: In Proceedings of the World Congress on Neural Networks, volume 1, pages 613–616, 1993.
BibTeX:
@inproceedings{martinez_93a,
author = {Martinez, Tony R. and Barker, J. Cory and Giraud-Carrier, Christophe},
title = {A Generalizing Adaptive Discriminant Network},
booktitle = {Proceedings of the World Congress on Neural Networks},
volume = {1},
pages = {613--616},
year = {1993},
}
Download the file: ps

Towards a General Distributed Platform for Learning and Generalization

Authors: Tony R. Martinez and Brent W. Hughes
Abstract: Different learning models employ different styles of generalization on novel inputs. This paper proposes the need for multiple styles of generalization to support a broad application base. The Priority ASOCS model (Priority Adaptive Self-Organizing Concurrent System) is overviewed and presented as a potential platform which can support multiple generalization styles. PASOCS is an adaptive network composed of many simple computing elements operating asynchronously and in parallel. The PASOCS can operate in either a data processing mode or a learning mode. During data processing mode, the system acts as a parallel hardware circuit. During learning mode, the PASOCS incorporates rules, with attached priorities, which represent the application being learned. Learning is accomplished in a distributed fashion in time logarithmic in the number of rules. The new model has significant learning time and space complexity improvements over previous models.
Reference: In Proceedings of the Conference on Artificial Neural Networks and Expert Systems ANNES’93, pages 216–219, 1993.
BibTeX:
@inproceedings{martinez_93b,
author = {Martinez, Tony R. and Hughes, Brent W.},
title = {Towards a General Distributed Platform for Learning and Generalization},
booktitle = {Proceedings of the Conference on Artificial Neural Networks and Expert Systems {ANNES}'93},
pages = {216--219},
year = {1993},
}
Download the file: ps

A Learning Model for Adaptive Network Routing

Authors: Tony R. Martinez and George L. Rudolph
Abstract: Increasing size, complexity, and dynamics of networks require adaptive routing mechanisms. This paper proposes initial concepts towards a learning and generalization mechanism to support adaptive real-time routing. An ASOCS learning model is employed as the basic adaptive router. Generalization of routing is based not only on source/destination address, but also on such factors as packet size, priority, privacy, network congestion, etc. Mechanisms involving continual adaptation based on feedback are presented. Extensions to conventional addressing which can support learning and generalization are proposed.
Reference: In Proceedings of the International Workshop on Applications of Neural Networks to Telecommunications IWANNT’93, pages 183–187, 1993.
BibTeX:
@inproceedings{martinez_93c,
author = {Martinez, Tony R. and Rudolph, George L.},
title = {A Learning Model for Adaptive Network Routing},
booktitle = {Proceedings of the International Workshop on Applications of Neural Networks to Telecommunications {IWANNT}'93},
pages = {183--187},
year = {1993},
}
Download the file: ps

A Survey of Neural Network Research and Fielded Applications

Authors: David Kemsley and Tony R. Martinez and Douglas M. Campbell
Abstract: This paper gives a tabular presentation of approximately one hundred current neural network applications at different levels of maturity, from research to fielded products. The goal of this paper is not to be exhaustive, but to give a sampling overview demonstrating the diversity and amount of current application effort in different areas. The paper should aid both researchers and implementors to understand the diverse and potential impact of neural networks in real world applications. Tabular information is given regarding different features of neural network application efforts including model used, types of input and output data, accuracy, and research status. An extended bibliography allows a mechanism for further study into promising areas.
Reference: International Journal of Neural Networks, volume 2/3/4, pages 123–133, 1992.
BibTeX:
@article{kemsley_92,
author = {Kemsley, David and Martinez, Tony R. and Campbell, Douglas M.},
title = {A Survey of Neural Network Research and Fielded Applications},
journal = {International Journal of Neural Networks},
volume = {2/3/4},
pages = {123--133},
year = {1992},
}
Download the file: ps

A Self-Adjusting Dynamic Logic Module

Authors: Tony R. Martinez and Douglas M. Campbell
Abstract: This paper presents an ASOCS (Adaptive Self-Organizing Concurrent System) model for massively parallel processing of incrementally defined rule systems in such areas as adaptive logic, robotics, logical inference, and dynamic control. An ASOCS is an adaptive network composed of many simple computing elements operating asynchronously and in parallel. This paper focuses on Adaptive Algorithm 2 (AA2) and details its architecture and learning algorithm. AA2 has significant memory and knowledge maintenance advantages over previous ASOCS models. An ASOCS can operate in either a data processing mode or a learning mode. During learning mode, the ASOCS is given a new rule expressed as a boolean conjunction. The AA2 learning algorithm incorporates the new rule in a distributed fashion in a short, bounded time. During data processing mode, the ASOCS acts as a parallel hardware circuit.
Reference: Journal of Parallel and Distributed Computing, volume 4, pages 303–313, 1991.
BibTeX:
@article{martinez_91a,
author = {Martinez, Tony R. and Campbell, Douglas M.},
title = {A Self-Adjusting Dynamic Logic Module},
journal = {Journal of Parallel and Distributed Computing},
volume = {4},
pages = {303--313},
year = {1991},
}
Download the file: ps

A Self-Organizing Binary Decision Tree for Incrementally Defined Rule Based Systems

Authors: Tony R. Martinez and Douglas M. Campbell
Abstract: This paper presents an ASOCS (adaptive self-organizing concurrent system) model for massively parallel processing of incrementally defined rule systems in such areas as adaptive logic, robotics, logical inference, and dynamic control. An ASOCS is an adaptive network composed of many simple computing elements operating asynchronously and in parallel. This paper focuses on adaptive algorithm 3 (AA3) and details its architecture and learning algorithm. It has advantages over previous ASOCS models in simplicity, implementability, and cost. An ASOCS can operate in either a data processing mode or a learning mode. During the data processing mode, an ASOCS acts as a parallel hardware circuit. In learning mode, rules expressed as boolean conjunctions are incrementally presented to the ASOCS. All ASOCS learning algorithms incorporate a new rule in a distributed fashion in a short, bounded time.
Reference: In IEEE Transactions on Systems, Man, and Cybernetics, volume 5, pages 1231–1238, 1991.
BibTeX:
@inproceedings{martinez_91b,
author = {Martinez, Tony R. and Campbell, Douglas M.},
title = {A Self-Organizing Binary Decision Tree for Incrementally Defined Rule Based Systems},
booktitle = {{IEEE} Transactions on Systems, Man, and Cybernetics},
volume = {5},
pages = {1231--1238},
year = {1991},
}
Download the file: ps

ASOCS: Towards Bridging Neural Network and Artificial Intelligence Learning

Authors: Tony R. Martinez
Abstract: A new class of connectionist architectures is presented called ASOCS (Adaptive Self-Organizing Concurrent Systems) [3,4]. ASOCS models support efficient computation through self-organized learning and parallel execution. Learning is done through the incremental presentation of rules and/or examples. Data types include Boolean and multi-state variables; recent models support analog variables. The model incorporates rules into an adaptive logic network in a parallel and self organizing fashion. The system itself resolves inconsistencies and generalizes as the rules are presented. After an introduction to the ASOCS paradigm, the abstract introduces current research thrusts which significantly increase the power and applicability of ASOCS models. For simplicity, we discuss only boolean mappings in the ASOCS overview.
Reference: In Proceedings of the 2nd Government Neural Network Workshop, 1991.
BibTeX:
@inproceedings{martinez_91c,
author = {Martinez, Tony R.},
title = {{ASOCS}: Towards Bridging Neural Network and Artificial Intelligence Learning},
booktitle = {Proceedings of the 2nd Government Neural Network Workshop},
year = {1991},
}
Download the file: ps

A Connectionist Method for Adaptive Real-Time Network Routing

Authors: Kelly C. McDonald and Tony R. Martinez and Douglas M. Campbell
Abstract: This paper proposes a connectionist mechanism to support adaptive real-time routing in computer networks. In particular, an Adaptive Self-Organizing Concurrent System (ASOCS) model is used as the basic network router. ASOCS are connectionist models which achieve learning and processing in a parallel and self-organizing fashion. By exploiting parallel processing the ASOCS network router addresses the increased speed and complexity in computer networks. By using the ASOCS adaptive learning paradigm, a network router can utilize more flexible routing algorithms.
Reference: In Proceedings of the 4th International Symposium on Artificial Intelligence, pages 371–377, 1991.
BibTeX:
@inproceedings{mcdonald_91,
author = {McDonald, Kelly C. and Martinez, Tony R. and Campbell, Douglas M.},
title = {A Connectionist Method for Adaptive Real-Time Network Routing},
booktitle = {Proceedings of the 4th International Symposium on Artificial Intelligence},
pages = {371--377},
year = {1991},
}
Download the file: ps

Smart Memory: The Memory Processor Model

Authors: Tony R. Martinez
Abstract: This paper overviews and proposes a class of smart memory devices called the memory processor model. Smart memory entails the tight coupling of memory and logic. The model seeks to alleviate the von Neumann bottleneck, take advantage of technology trends, improve overall system speed, and add encapsulation advantages. Speed is increased through locality of processing, communication savings, higher-level functionality, and parallelism. Parallelism is exploited at both the micro and macro levels. Data objects are accessed through descriptors, which give the memory a meta-knowledge concerning the objects, allowing for nontraditional access mechanisms. Both data types and operations are programmable. Innovative processing schemes, coupled with emerging technology densities, allow for substantial speed-up in traditional and novel memory operations. Three important paradigms introduced are descriptor processing, where operations are accomplished without access to the actual data, associative descriptor processing, supporting highly parallel access and processing, and the single-program multiple-data method, allowing parallelism by simultaneous processing of data objects distributed amongst multiple smart memories. Examples of specific operations are presented. This papers presents initial studies into the smart memory mechanism with the goal of describing its potential and stimulating further work.
Reference: In IFIP International Conference, 1990. In Modeling the Innovation: Communications, Automation and Information Systems, Carnevale, Lucertini, and Nicosia (Eds), North-Holland, pages 481–488, 1990.
BibTeX:
@inproceedings{martinez_90a,
author = {Martinez, Tony R.},
title = {Smart Memory: The Memory Processor Model},
booktitle = {{IFIP} International Conference, 1990. In Modeling the Innovation: Communications, Automation and Information Systems, Carnevale, Lucertini, and Nicosia (Eds), North-Holland},
pages = {481--488},
year = {1990},
}
Download the file: ps

Consistency and Generalization of Incrementally Trained Connectionist Models

Authors: Tony R. Martinez
Abstract: This paper discusses aspects of consistency and generalization in connectionist networks which learn through incremental training by examples or rules. Differences between training set learning and incremental rule or example learning are presented. Generalization, the ability to output reasonable mappings when presented with novel input patterns, is discussed in light of the above learning methods. In particular, the contrast between hamming distance generalization and generalizing by high order combinations of critical variables is overviewed. Examples of detailed rules for an incremental learning model are presented for both consistency and generalization constraints.
Reference: In Proceedings of the International Symposium on Circuits and Systems, pages 706–709, 1990.
BibTeX:
@inproceedings{martinez_90b,
author = {Martinez, Tony R.},
title = {Consistency and Generalization of Incrementally Trained Connectionist Models},
booktitle = {Proceedings of the International Symposium on Circuits and Systems},
pages = {706--709},
year = {1990},
}
Download the file: ps

Progress in Neural Networks, ch. 5

Authors: Tony R. Martinez
Abstract: (Book chapter that overviews the first three ASOCS models.)
Reference: Omidvar, Omid, editor, volume 1, chapter Adaptive Self-Organizing Concurrent Systems, pages 105–126, 1990. Ablex Publishing.
BibTeX:
@inbook{martinez_90c,
author = {Martinez, Tony R.},
title = {Progress in Neural Networks, ch. 5},
editor = {Omidvar, Omid},
volume = {1},
chapter = {Adaptive Self-Organizing Concurrent Systems},
pages = {105--126},
year = {1990},
note = {Ablex Publishing},
}
Download the file: ps

Smart Memory Architecture and Methods

Authors: Tony R. Martinez
Abstract: This paper discusses potential functionalities of smart memories. Smart memory entails the tight coupling of memory and logic. A specific architecture called the memory processor model is proposed. The model seeks to alleviate the von Neumann bottleneck, take advantage of technology trends, improve overall system speed, and add encapsulation advantages. Speed is increased through locality of processing, communication savings, higher-level functionality, and parallelism. Data objects are accessed through descriptors, which give the memory a meta-knowledge concerning the objects, allowing for nontraditional access mechanisms. Both data types and operations are programmable, and the model is streamlined for memory operations and services. Innovative processing schemes, coupled with emerging technology densities, allow for substantial fine-grain parallelism in traditional and novel memory operations. Three important paradigms introduced are descriptor processing, where operations are accomplished without access to the actual data, associative descriptor processing, supporting highly parallel access and processing, and the single-program multiple-data method, allowing parallelism by simultaneous processing of data objects distributed amongst multiple smart memories. Examples of specific operations are presented. This paper presents initial studies into the smart memory mechanism with the goal of describing its potential and stimulating further work.
Reference: Future Generation Computer Systems, volume 6, pages 145–162, 1990.
BibTeX:
@article{martinez_90d,
author = {Martinez, Tony R.},
title = {Smart Memory Architecture and Methods},
journal = {Future Generation Computer Systems},
volume = {6},
pages = {145--162},
year = {1990},
}
Download the file: ps

On the Pseudo Multilayer Learning of Backpropagation

Authors: Tony R. Martinez and M. Lindsey
Abstract: Rosenblatt’s convergence theorem for the simple perceptron initiated much excitement about iterative weight modifying neural networks. However, this convergence only holds for the class of linearly separable functions, which is vanishingly small compared to arbitrary functions. With multilayer networks of nonlinear units it is possible, though not guaranteed, to solve arbitrary functions. Backpropagation is a method of training multilayer networks to converge to the solution of arbitrary functions. This paper describes how classification takes place in single and multilayer networks using threshold or sigmoid nodes. It then shows that the current backpropagation method can only do effective learning on one layer of a network at a time.
Reference: In Proceedings of the IEEE Symposium on Parallel and Distributed Processing, pages 308–315, 1989.
BibTeX:
@inproceedings{martinez_89a,
author = {Martinez, Tony R. and Lindsey, M.},
title = {On the Pseudo Multilayer Learning of Backpropagation},
booktitle = {Proceedings of the {IEEE} Symposium on Parallel and Distributed Processing},
pages = {308--315},
year = {1989},
}
Download the file: ps

Neural Network Applicability: Classifying the Problem Space

Authors: Tony R. Martinez
Abstract: The tremendous current effort to propose neurally inspired methods of computation forces closer scrutiny of real world application potential of these models. This paper categorizes applications into classes and particularly discusses features of applications which make them efficiently amenable to neural network methods. Computational machines do deterministic mappings of inputs to outputs and many computational mechanisms have been proposed for problem solutions. Neural n